Measuring the coherence of writing using topic-based analysis

Measuring the coherence of writing using topic-based analysis Abstract Among the many possible aspects to assess in writing, one of the most problematic is coherence. The problems with marking coherence arise because it is by nature subjective. However, the reasonable probability of several readers reaching a consensus concerning the coherence of a text suggests that it may be possible to assign relatively reliable marks for coherence. In this paper, topic-based analysis (Watson Todd, 1998) was chosen as a method of evaluating coherence and the results of this evaluation were compared with teachers' marks for coherence. Topicbased analysis involves identifying key concepts in a text, identifying the relationships between these concepts, linking the relationships into a hierarchy, and mapping the text onto the hierarchy. Doing this allows several different measures of coherence to be generated. Comparing these measures against the teachers' scores, it was found that the number of moves between key concepts per ten T-units correlated most closely with the teachers' marks. The results have implications concerning the basis of teachers' marks for coherence and the applicability of topic-based analysis as a method of assessing coherence of writing.